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Abstract 

Background: Studies have shown that communities have not always been able to implement evidence-based 
prevention programs with quality and achieve outcomes demonstrated by prevention science. Implementation 
support interventions are needed to bridge this gap between science and practice. The purpose of this article is to 
present two-year outcomes from an evaluation of the Assets Getting To Outcomes (AGTO) intervention in 12 Maine 
communities engaged in promoting Developmental Assets, a positive youth development approach to prevention. 
AGTO is an implementation support intervention that consists of: a manual of text and tools; face-to-face training, 
and onsite technical assistance, focused on activities shown to be associated with obtaining positive results across 
any prevention program. 

Methods: This study uses a nested and cross-sectional, cluster randomized controlled design. Participants were 
coalition members and program staff from 12 communities in Maine. Each coalition nominated up to five 
prevention programs to participate. At random, six coalitions and their respective 30 programs received the 
two-year AGTO intervention and the other six maintained routine operations. The study assessed prevention 
practitioner capacity (efficacy and behaviors), practitioner exposure to and use of AGTO, practitioner perceptions of 
AGTO, and prevention program performance. Capacity of coalition members and performance of their programs 
were compared between the two groups across the baseline, one-, and two-year time points. 

Results: We found no significant differences between AGTO and control group's prevention capacity. However, 
within the AGTO group, significant differences were found between those with greater exposure to and use of 
AGTO. Programs that received the highest number of technical assistance hours showed the most program 
improvement. 

Conclusions: This study is the first of its kind to show that use of an implementation support intervention-AGTO -yielded 
improvements in practitioner capacity and consequently in program performance on a large sample of practitioners and 
programs using a randomized controlled design. 
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Background 

Prevention programming can improve youth outcomes 
and be cost effective [1]. While programs need to be well 
implemented, they also require practices such as tracking 
outcomes and engaging in continuous improvement-to 
reap these benefits [2]. Yet, many studies have shown that 
communities have not been able to implement evidence- 
based prevention programs with such quality and achieve 
outcomes demonstrated by prevention science [3-8]. This 
'gap' between science and practice (e.g., [9,10]) can result 
when communities lack individuals with the capacity- 
defined as the self-efficacy and behaviors -needed to engage 
in critical prevention practices. 

Getting To Outcomes® (GTO @ ) a was developed to 
bridge this gap between science and practice. GTO is an 
implementation model (Figure l)-specifying ten steps 
(or sets of activities) prevention practitioners should 
take, shown to be associated with obtaining positive re- 
sults across many different prevention programs [11]. The 
first six involve planning activities (needs assessment, goal 
setting, choosing programs, ensuring appropriate capacity 
and fit, planning program details), the next two steps 
are process and outcome evaluation [7,8], and the last 
two steps involve using data to improve programs and 
sustaining programs [9,10]. GTO is also an implementation 
support intervention, which strengthens the efficacy and 
behaviors (collectively known as capacity) community prac- 
titioners need to perform these activities with quality. GTO 



has been found to improve the capacity of individual practi- 
tioners and the performance of drug prevention programs 
in both quasi-experimental [12] and randomized controlled 
trials [13]. Congruent with social cognitive theories of be- 
havioral change [14-17], the efficacy and behaviors related 
to the activities targeted by GTO s 10 steps (i.e., prevention 
capacity' at the individual level), are related to how well 
prevention is carried out at the program level [18], which, 
in turn, affects outcomes. 

GTO has been adapted to a number of content areas, 
recently to the model of positive youth development devel- 
oped by Search Institute. This model, called Developmental 
Assets/ states that in order to thrive, young people need 40 
key developmental supports and experiences that include 
but are broader than the notion of protective factors. From 
a caring school climate to participation in high-quality 
after-school programs, these factors have been shown 
to predict health and well-being outcomes consistently 
across sex, race-ethnicity, and family income [19-21]. This 
approach was blended into the 10 GTO steps, so that each 
step was enhanced to include a specific focus on building as- 
sets. For example, in step one (choose which problem(s) to 
focus on) individuals use both asset indicators, as well as risk 
information, to help select priorities. This blended approach 
is called Assets-Getting To Outcomes (AGTO) [22]. 

The AGTO model and corresponding intervention 
is grounded in implementation theory. For example, 
AGTO operationalizes the Consolidated Framework 
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for Implementation Research (CFIR) to ensure that all 
the major domains that influence implementation are 
considered [23]. Intervention characteristics (e.g., evidence 
strength and quality, relative advantage, complexity, 
adaptability), the first domain, can influence whether 
practitioners adopt an intervention. Without adaptation, 
many interventions come to settings as a poor fit. Even if 
these interventions have been known to improve outcomes, 
practitioners do not always adopt these interventions, 
underscoring the need for core intervention elements and 
what the CFIR terms an adaptable intervention periphery 
(i.e., elements that can be adapted to improve intervention 
fit [3]. The AGTO intervention walks practitioners through 
a systematic process to assess, and then improve, how well 
interventions fit with their target population, organization, 
and broader community (step four: Fit). 

The next three domains of the CFIR comprise the 
outer setting (e.g., broader social, political and economic 
context including policies, incentives, resources etc.), 
inner setting in which the intervention is implemented 
(e.g., context of the specific organization or group 
implementing the intervention including the structural 
characteristics, relationships, implementation readiness), 
and the characteristics of the individuals involved (e.g., 
knowledge, skills). Attention needs to be paid to fac- 
tors both at the individual and organizational levels 
that impact the degree to which new practices are 
adopted and implemented. For example, a number of 
studies have shown that factors at both the individual level 
(e.g., training, skills, efficacy, involvement in decision mak- 
ing, and satisfaction of teachers) and at the organizational 
level (organization size, climate, and financial resources, 
and active support of principals and administrators) 
have been shown to predict successful implementation 
of prevention in public schools [3,24-28]. In the AGTO 
study, the inner setting is the coalition, the outer setting 
is the broader community where coalitions are located, 
and the individuals involved are the prevention practi- 
tioners who are coalition members. To create both cap- 
acity and conditions for successful implementation of 
programs, the AGTO intervention targets both practi- 
tioners and coalitions. While AGTO helps the inner set- 
ting and individuals account for the outer setting, it is 
beyond its scope to specifically alter the broader social, 
political, and economic context. 

Implementation process or the active change process 
aimed to achieve individual and organization use of the 
intervention as designed is the final domain in the CFIR. 
Research suggests that innovative multi-level strategies 
are needed to improve the implementation process. 
Passive approaches such as trainings by themselves do not 
lead to change, as attendees often experience barriers to 
incorporating newly learned information into their work 
[29,30]. The AGTO intervention, which is proactive versus 



passive, specifically targets improvements to this implemen- 
tation process at both the individual and program level so 
that implementation more closely aligns with empirically- 
based, high-quality prevention processes. 

The purpose of this article is to present two-year 
outcomes from an evaluation of AGTO in 12 Maine 
communities engaged in promoting Developmental 
Assets (interim one year outcomes have been presented 
elsewhere [18]). At random, six coalitions and their respect- 
ive programs received the two-year AGTO intervention 
and the other six maintained routine operations. Capacity 
of coalition members and performance of their programs 
were compared between the two groups across the baseline, 
one-, and two-year timepoints. The study had the following 
hypotheses: 

Hypothesis one: Individual prevention practitioners in 
the AGTO group would demonstrate greater gains in 
capacity than those in the control group. 

Hypothesis two: Individual prevention practitioners 
with greater exposure to and use of AGTO would dem- 
onstrate greater gains in capacity than those who were 
assigned to use AGTO but used less or none. 

Hypothesis three: Whole prevention programs in the 
AGTO group would demonstrate significant gains in 
prevention performance (i.e., performing various preven- 
tion tasks with high quality) overall and in proportion with 
their use of AGTO. 

Building upon previous GTO studies, the current 
study is the first to evaluate GTO on both individual 
capacity and program performance in a randomized trial 
with a larger number of programs in a way that uses a 
traditional approach to conducting a randomized trial, 
while being flexible enough to account for the complex, 
interacting, multi-level, and transients states of constructs 
in the real world' [31]. Consistent with CFIR, the study 
attempts to evaluate AGTO in terms of its impact on 
three of the CFIR domains: inner setting, individual 
characteristics, and implementation process; while using 
measures of the outer setting as covariates. Table 1 provides 
a summary of how the study design aligns with and 
measures the CFIR domains. 

Methods 

Study design 

The study was a nested cross-sectional matched pairs 
group randomized trial [32]. Twelve coalitions were 
randomized to either receive the two-year AGTO 
intervention or to serve as comparison coalitions. Prior 
to randomization, coalitions were matched into pairs 
based on the total population and demographic charac- 
teristics of the community served by each coalition 
(from the 2000 Census) and a rating of each coalition s 
functioning at baseline. Two technical assistance pro- 
viders and a local coalition expert came to consensus 
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Table 1 CFIR domains by study design and measures 



CFIR domains 


How the domain aligns with AGTO 


Relevant study measures 


Intervention characteristics 


AGTO walks practitioners through a systematic process to 
assess, and then improve, how well interventions fit with their 
target population, organization, and broader community 


No study-wide measures, but AGTO supported the 
development of process measures to assess these 
characteristics at the program level 


Outer setting 


Broader social, economic, and political context of the 12 

nsrtirinstinn rnmmi initip^ ni i"KiHp thp <;rnnp nf AGTO 

kJUl IIV.ILJGLII l^A {~{J\ 1 II 1 IUI IIIICO, UULjIUC LI ICT D^-WkJC: Ul / \VJ 1 w 


Practitioners perceptions of how community context 

sffprtpH nrnnrsm imnlpmpntstinn fm islitstivp intprvipwO 

Q 1 1 ICTH kJI IQI 1 1 II 1 IkJICrl 1 IC7I ILULI^JI 1 \{J UUI 1 LU LI VC III LCTI V ICT V v ~>J 


Individuals involved 


AGTO builds the prevention capacity of coalition members and 
program staff that comprise the study sample 


Practitioners prevention self-efficacy with AGTO and 
AGTO behaviors (coalition survey) 


Inner setting 


AGTO targets improving the setting of interventions within the 
12 community-based coalitions and 60 programs that comprise 
the study sample 


Practitioners satisfaction with coalition membership 
and leadership (coalition survey) 


Implementation process 


AGTO proactively targets improvements to the implementation 
process at both the individual and program level so that 
implementation more closely aligns with empirically-based 
high quality prevention processes 


Self-reported exposure to AGTO (coalition survey) and 
utilization of AGTO TA (TA providers log) Program 
implementation along the 10 steps of AGTO 
(capacity interview) 



to derive the pairings based on each participating coa- 
litions overall stage of development, as well as their 
stage of development in four functional areas: coalition 
development and management, coordination of prevention 
programs and services, conducting environmental strat- 
egies, and acting as intermediary or support organization to 
other community organizations and coalitions. Coalitions 
were rated on their mastery of each functional area on a 
Likert scale. The matched pairs were sent to the project 
statistician, who allocated one coalition per matched pair 
to the AGTO intervention by using the random number 
generator in Microsoft Excel. Programs and practitioners 
followed their coalition assignment. The rating and 
resulting four-stage categorization were developed by 
Office of National Drug Control Policy's evaluation of 
the Drug-Free Communities program after reviewing 
the literature on coalition functioning and receiving 
input from experts and coalition leaders [33]. Analysis 
of these stages within the Drug-Free Communities program 
showed that the more advanced coalitions had stronger 
prevention practices and better outcomes [33]. As will be 
seen below, it was not possible to blind research staff or 
participants to their study condition. 

Setting: 12 community-based coalitions 

Participants were all the coalition members and program 
staff from 12 coalitions in Maine. Community-based 
coalitions are comprised of prevention practitioners 
from non-profit organizations from multiple levels 
(individual, organizational, policy) and sectors (parents, 
youth, criminal justice, education) that collaborate to 
improve community health [34]. These 12 coalitions are 
similarly structured (a small group of paid staff working with 
a larger group of volunteers), work in similar geographic 
and demographics settings, 13 and have similar rates of 
reported youth risk behaviors (e.g., alcohol and drug use), c 
missions (to promote healthy youth development), annual 
budgets, and numbers of distinct programs. However, it is 



important to note that membership and leadership in coali- 
tions are not static, but change over time. These coalitions 
represent the inner setting defined by the CFIR. 

Participants 
Prevention practitioners 

Coalition members, characterized by the CFIR as the in- 
dividuals involved, completed a coalition survey at base- 
line (just prior to the AGTO intervention, April 2009), 
mid (about one year later, June 2010) and post (about 
two years later, July 2011). After randomization, but 
prior to completing the survey, we received approval 
from RANDs Human Subjects Protection Committee 
for the study and written consent was obtained from all 
participants. The overall response rate was 82% across all 
sites at baseline, 79% at mid, and 89% at post. Community 
coalitions often experience high turnover and the same 
was true here, as 41% percent of those surveyed at base- 
line or mid were also surveyed at post. New participants 
joined the coalition after baseline as well. There were 376 
survey participants at baseline, 303 at mid, and 315 at post 
(152 of whom had participated in the baseline survey 
and 163 who had not), yielding a total of 591 unique 
participants. The size of the clusters used in the analyses 
(in this case, coalitions) ranged from n = 8 to n = 50 
across the 12 coalitions and three time points. In lieu of 
a CONSORT diagram (Additional file 1), Table 2 presents 
additional response rate and sample size information is, 
which better presents this information for this study. At 
baseline and post, we compared the distributions of several 
participant characteristics (see covariates below) between 
the intervention and control groups using ordinal, logistic, 
or linear regression, with the AGTO group indicator as a 
predictor variable (see Table 3). We found no statistically 
significant differences in characteristics between the groups 
at either baseline or post, except for satisfaction with in- 
volvement in the coalition. The AGTO group reported sig- 
nificantly higher satisfaction at post than the comparison 
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Table 2 Additional sample size information 

Response rates, % Use of AGTO, % Sample sizes 

Coalition Study Baseline Mid Post First year Second year Baseline Mid Post All time- Baseline Mid and post Baseline 
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2a 


Control 


70.83 


61.90 


81.25 


18 


23 


17 


13 


13 


18 


10 






2b 


AGTO 


81.82 


86.67 


93.55 


47 


50 


17 


26 


29 


39 


6 


10 




3a 


Control 


80.65 


81.40 


92.86 


12 


17 


50 


35 


39 


54 


14 


8 


2 


3b 


AGTO 


83.87 


80.39 


84.62 


34 


49 


50 


41 


33 


60 


14 


12 




4a 


Control 


95.24 


78.05 


83.33 


23 


32 


40 


31 


30 


45 


16 


14 




4b 


AGTO 


80.00 


72.73 


88.89 


21 


47 


24 


15 


24 


27 


2 






5a 


Control 


74.36 


74.36 


82.98 


17 


24 


29 


29 


39 


42 


10 


12 




5b 


AGTO 


87.18 


85.00 


95.12 


32 


38 


34 


34 


39 


60 


4 


14 




6a 


Control 


79.55 


70.59 


100.00 


9 


25 


35 


12 


11 


12 


12 






6b 


AGTO 


87.80 


89.74 


88.46 


20 


24 


35 


33 


23 


54 


12 


6 





group (p = 0.003). Analyses were adjusted to account for 
this statistically significant difference, and given that it was 
one of a large number of t-tests we do not anticipate this 
difference influenced the findings presented in this article. 

Prevention programs 

Each coalition nominated up to five prevention programs 
to participate in the study (one-half of the programs 
were assigned to intervention group, one-half to the 
control group). Despite some variability, all programs 
aimed to promote healthy development with middle and 
high school youth. Although some were evidence-based 
programs including Project ALERT [35], Lifeskills Training 
[36], and Making Proud Choices [37-39], most were 
designed by coalitions, including mentoring, social norms 
campaigns, programs to divert first offenders from the ju- 
venile justice system, and leadership training. Most pro- 
grams were ongoing at the outset of the study. 

AGTO intervention 

The AGTO intervention includes three types of assistance 
which are adapted to fit the needs and priorities of the 
individuals involved, as well as the inner and outer setting: 
a manual of text and tools; face-to-face training, and 
onsite technical assistance (TA). These three types of 
assistance aim to improve the implementation process for 
each program. Two full-time, Maine-based staff, one with 
a masters and one with a bachelors degree, provided 
AGTO tools, training, and TA to the intervention coalitions 
and programs during the two-year intervention period. 
The tools are in the Search Institute -published manual, 
Getting To Outcomes with Developmental Assets: Ten steps 
to measuring success in youth programs and communities 
[22], which all intervention participants received. The train- 
ing was delivered separately to each coalition over a full day 



after baseline, and covered the AGTO model, tools in the 
manual, and an introduction to the TA process. Based on 
TA literature [40-43], the AGTO-based TA involves three 
structured steps, including an initial diagnosis of program 
functioning, development of a logic model, and develop- 
ment of a plan for how the TA and program staff were to 
make improvements, carried out during and in between bi- 
weekly TA visits. TA staff provided consultation and feed- 
back to practitioners on conducting AGTO tasks as applied 
to their program [44-46]. An expanded discussion about 
the AGTO intervention is available elsewhere [18]. 

Measures 
Covariates 

The coalition survey included demographic information 
about prevention practitioners (i.e., age, gender, race/ 
ethnicity, education, and employment status), consistent 
with CFIR's individual characteristics domain. Measures 
related to coalition functioning, leadership, and the in- 
corporation of new practices were also included in the 
coalition survey, and relates to the inner setting domain 
of CFIR. The leadership score [47] is the mean of ten 
items rating the effectiveness of coalition leadership 
from five response options (1 = 'poor' to 5 = 'excellent 0, 
a B L = 0.96, a mid = 0.95, a post = 0.96. From the staff survey 
of organizational readiness for change measure [48], 
Cohesion (a BL = 0.86, a mid = 0.82, a post = 0.86) and recep- 
tivity to change scores (a B L = 0.73, a mid = 0.72, a post = 0.75) 
were adapted, averaging across five response options 
(1 = strongly disagree' to 5 = strongly agreed Stand-alone 
coalition survey items used in the analysis were overall 
satisfaction with involvement in the coalition (1 = Very 
dissatisfied' to 7 = Very satisfied'), years of involvement in 
the coalition, and type of involvement (paid staff, volunteer 
individual, or volunteer from a partner organization). 



Acosta et al. Implementation Science 2013, 8:87 
http://www.implementationscience.eom/content/8/1/87 



Page 6 of 16 



Table 3 Characteristics of study sample, AGTO versus non-AGTO h 







Baseline 




Post 




AGTO (n = 174) 


Non-AGTO (n = 202) 


AGTO (n = 159) 


Non-AGTO (n = 156) 


Age in years, % 


12-49 


53 


53 


46 


48 


50+ 


47 


47 


55 


52 


Female, % 


72 


74 


71 


74 


White, % 


98 


97 


97 


100 


Education, % 


HS graduate or less 


6 


12 


4 


6 


Some college 


22 


6 


21 


10 


College graduate 


21 


27 


27 


32 


Graduate education 


50 


54 


49 


51 


Employment status, % 


Full time 


81 


78 


78 


81 


Part time 


13 


9 


13 


12 


Out of labor force 


6 


12 


10 


7 


Coalition leadership (0-100), mean (SD) 


77.4 (16.1) 


71.9 (19.1) 


80 (17.7) 


74 (18.8) 


Coalition cohesion (0-100), mean (SD) 


76.0 (15.4) 


71.9 (14.3) 


77 (13.4) 


72 (14.8) 


Coalition receptivity to change (0-100), mean (SD) 


72.9 (13.0) 


69.1 (13.7) 


74 (11.5) 


69 (13.4) 


Satisfaction with involvement in coalition, % 


1 = Very dissatisfied, 2, 3 


6 


9 


2 


6 


4 


9 


16 


7 


19 


5 


17 


16 


13 


19 


6 


41 


43 


44 


46 


7 = Very satisfied 


27 


17 


34 


11 


Years in coalition, % 


0 


7 


13 


9 


21 


1 


25 


25 


8 


2 


2 


16 


16 


14 


20 


3 


7 


10 


15 


13 


4 


5 


13 


9 


11 


5 


8 


6 


8 


10 


6+ 


32 


18 


36 


24 


Type of involvement, % 


Paid staff 


31 


23 


26 


24 


Volunteer individual 


21 


33 


32 


34 


Volunteer from partner organization 


49 


45 


42 


42 



h There were no statistically significant differences between AGTO and non-AGTO study participants on almost all of these characteristics at baseline or post 



(p>0.05), except for satisfaction with involvement in coalition. 



Practitioner prevention capacity 

Assessed in the coalition survey, prevention capacity was 
defined as efficacy and behaviors of practitioners and relates 
to CFIR s individual characteristics domain. There are 
two scales that assess perceived efficacy (i.e., completing 
various tasks without assistance) using a three-point scale 
(1 = would need a great deal of help to carry out this task/ 

2 = could carry out this task, but would need some help/ 

3 = could carry out this task without any help). The asset 
efficacy scale includes five items asking about helping 



adults connect with youth; involving multiple sectors of the 
community; engaging youth in asset-building; incorporating 
assets into existing programs; and helping community 
leaders understand how their decisions affect the develop- 
ment of youth (a BL = 0.81, a mi d = 0.86, a post = 0.82). The 
GTO efficacy scale includes items asking about activities 
from the GTO steps -e.g., evaluation, planning, quality 
improvement (a BL = 0.84, a mid = 0.89, a post = 0.88). 

There are three behavior scales that are means of items 
with seven-point scales (1 = 'never' to 7 = very often") 
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assessing the frequency with which respondents engaged 
in these activities during the previous 12 months. The 
Asset Behaviors scale includes five items assessing 
whether individuals are motivating both adults and 
youth to become asset-builders; incorporating asset 
building into existing youth programs; influencing 
community leaders to implement asset- aligned policies 
(e.g., employ funding criteria that rewards asset-building); 
and engaging various sectors of the community to support 
asset building (a B L = 0.91, a mid = 0.93, a post = 0.93). GTO 
behaviors include 11 items that align with the 10 steps 
(two items focus on needs and resources assessment) 
(a BL = 0.92, a mid = 0.94, a post = 0.95). AGTO behaviors 
includes 11 items that tailor the GTO 10 steps to Assets 
(e.g., needs assessment on youth assets) (a BL = 0.94, 
a m id = 0.96, a post = 0.95). The GTO scales are shortened 
versions of scales used in previous studies (e.g., [12]). 
All Assets scales were developed for the current study, 
based on the structure used for the GTO scales. Although 
the goal of the intervention was build practitioners' capacity 
related to the blended AGTO model, GTO and assets 
efficacy and behaviors were asked about separately to 
identify which aspects of their capacity improved and 
to avoid masking potential effects of the intervention 
(e.g., practitioners that apply GTO to other areas of their 
work life, may have disproportionately greater increases in 
capacity related to GTO). 

Use of and exposure to AGTO 

To assess AGTOs potential impact on each programs 
implementation process, we first documented the degree 
to which each program engaged in the AGTO intervention 
through the AGTO participation index, which is the sum 
of six true/false items added to the mid and post coalition 
survey. Based on the Hall et al. model of categorizing the 
degree to which individuals use' an innovation [49-51], 
these items assess key markers of use including participa- 
tion in training, reading the materials, planning, discussing 
the model with colleagues, securing resources, and receiv- 
ing TA. Exposure to AGTO was also documented by TA 
providers recording hours of TA they delivered to each 
program, by AGTO step. The Participation Index and 
hours of TA have been shown to be related to prevention 
capacity and performance in a previous study of GTO 
[12,52]. A dichotomous measure was created if a user par- 
ticipated (AGTO participation index >1) at either mid or 
post. Most users who participated at mid also participated 
at post; however, there were a few cases in which respon- 
dents were documented to have used AGTO at mid but 
not at post (n = 4), or used at post, but not at mid (n = 19). 
Users at mid but not at post were assumed to hold residual 
knowledge or exposure at post. Preliminary analysis showed 
no significant differences between outcome measures of 
these groups and thus we felt confident characterizing 



participation as a single dichotomous measure of ever 
using (AGTO participation index >1) versus never using 
(AGTO participation index = 0). 

Program performance 

A structured interview was used to assess the impact of 
AGTO on the implementation process, administered on 
the same timeline as the coalition survey. Prevention practi- 
tioners performance of tasks associated with high-quality 
prevention targeted by AGTO were captured through 
the interview. Whole programs are rated, not individuals, 
because programs operate as a unit. Using the interview re- 
sponses, a set of ratings were made assessing performance 
of activities of seven key domains: goals and objectives, best 
practices, planning, process evaluation, outcome evaluation, 
continuous quality improvement, and sustainability. Be- 
cause programs were mid-implementation the other three 
AGTO domains (needs assessment, fit, and capacity), all of 
which are pre-program implementation activities, were not 
assessed. The ratings come from 10 items (or components') 
that assess how well each of the abovementioned activities 
are performed over the last year. Each component has 
seven response choices, described with specific behaviors, 
that range from highly faithful = 7' to highly divergent = V 
from ideal performance. This measure has been shown 
to be sensitive to change and reliable in previous GTO 
studies [12,53]. We calculated a score for each of the 10 
components and a total score (i.e., average across all 
10 components) for both the intervention and control- 
assigned programs at baseline and post. We then calculated 
the percent change for each component and the total score 
between baseline and post for all programs. In addition to 
the ratings, it was simply noted for each program whether 
they had conducted any process and outcome evaluation at 
each time point (yes/no). 

Trained research staff conducted and audiotaped the 
structured interviews by telephone with the directors of 
51 programs at baseline. The 32 d programs that were 
still operating two years later were interviewed again at 
post (AGTO = 17, control = 15). A second coder double- 
rated 10% of the programs using the audio recordings. The 
two raters discussed the scoring to reach a consensus when 
there were discrepancies. We calculated percent agreement 
and estimated a prevalence-adjusted bias-adjusted kappa 
(PABAK) [54] because of the potential impact on kappa 
from the relatively high prevalence of positive ratings. 
At baseline and post, respectively, the percent agree- 
ment between raters was 79% and 99% and the PABAK 
was 0.59 and 0.99. 

Perceptions of AGTO 

After the intervention concluded, we conducted semi- 
structured interviews with staff from 13 programs 
(March to May, 2012) to gather practitioners' perceptions 
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of AG TO, focusing on ease of use and satisfaction with 
AGTO as well as the benefits and challenges they ex- 
perienced with AGTO. These are all characteristics 
that influence implementation success and are consistent 
with the intervention characteristics identified by the 
CFIR. A convenience sample of staff that were consistently 
engaged in the project (i.e., no turnover) and significant 
users of AGTO were selected for these interviews because 
they were determined to be those most likely to have the 
knowledge and experience needed for the interview. 

Statistical analysis 

Assessing changes in prevention capacity 

We used two approaches to assess changes in capacity: 
an intent-to-treat analysis and because AGTO is a volun- 
tary intervention, an analysis comparing users of AGTO 
to non-users. Our data was clustered by individuals within 
matched coalition, therefore, for the intent-to-treat analysis 
we fit a model with two sets of random effects, one for 
matched coalitions and one with random intercepts and lin- 
ear functions of time to account for trajectories of repeated 
observations within respondent. The model included three 
fixed effect terms: group (AGTO versus control), time 
(baseline = 0 years, mid = 1 year, post = 2 years), and an 
interaction between group and time. The interaction term 
(i.e., (difference of differences') was tested to determine 
whether the slope of the best-fit line across all three 
time points varied by group. The data were analyzed in 
accordance with intervention assignment. For the AGTO 
user versus non-user analysis, we repeated the intent- 
to-treat models, but instead of comparing the AGTO 
to non-AGTO groups we compared AGTO users 
(AGTO participation index >1) to AGTO non-users 
(Index = 0). To ensure these groups were equivalent we 
conducted preliminary analyses to determine if there were 
significant differences in baseline capacity measures. Only 
one significant difference was detected in GTO behaviors 
(p = 0.047, SE = 3.629), and given that it was one of a large 
number of t-tests we do not anticipate this difference 
influenced the findings presented in this article. 

Assessing use of and exposure to AGTO 

Means and percentages were calculated for the partici- 
pation index, and its individual components, for both 
study groups. 

Program performance and its relationship to AGTO 
exposure 

Given the small number of programs in this analysis, we 
were limited to primarily descriptive analyses. To assess 
the relationship between the exposure to AGTO (i.e., TA 
hours, by performance domain) and change in program 
performance, we aggregated across all AGTO programs 
and treated each AGTO domain as a case (i.e., n = 7), 



and calculated the relationship of TA hours spent on 
each domain with the amount of change exhibited in 
that domain s Performance rating. 

Assessing perceptions of AGTO 

We used constant comparative analysis to analyze 
structured interview data [55]. A member of the research 
team read each interview transcript and coded text to 
identify categories representing program staff percep- 
tions of AGTO. Thirty-eight percent of transcripts 
were read and coded by two different team members. 
Inter-rater reliability was 0.86. After coding, the codes 
were sorted and categorized into themes. Thirty-one 
themes (e.g., It is easier to implement AGTO when it is 
integrated into routine activities) were identified and 
grouped into categories (e.g., factors that facilitated the 
implementation of AGTO). 

Results 

Prevention capacity of AGTO versus control 

Model-based mean estimates at each of the time points 
are shown in Table 4 along with tests of the change over 
time within group and between groups. In the intent- 
to-treat analysis, assets efficacy within the control group 
significantly increased over time (a 2.6 point increase, 
SE = 1.23) between each time point, making a 5.2 increase 
from baseline to post; p = 0.035. The control group showed 
a decline in GTO behaviors (adjusted averaged decline 
of 4.3 points baseline to post) but the change was not 
significant (p = 0.055). The amount of change between 
the intervention and control groups did not significantly 
differ across the three time points for assets efficacy, 
GTO behaviors, or for any of the other prevention capacity 
scales in tests of condition by time interaction. 

The clustering among the five outcomes was low. It 
was 0.00 for assets efficacy (95% CI, 00.0, 0.09) and 
GTO behaviors (95% CI, 00.0, 0.07), 0.01 for GTO effi- 
cacy (95% CI, 00.0, 0.09) and assets behaviors (95% CI, 
00.0, 0.07), and 0.02 for AGTO behaviors (95% CI, 00.0, 
0.07). All clusters were represented in the analyses. 

Use of and exposure to AGTO 

The participation index showed that 47% of those in the 
AGTO group (N = 281) had used some portion of the 
intervention (participation index >1, M = 1.94, SD = 2.37) 
by reading AGTO materials (37%), participating in train- 
ing (36%), talking to others about AGTO (33%) or making 
plans to use it (32%), receiving TA (30%), or securing 
resources for AGTO (24%). For the non-AGTO group, 
participation was at 20.6% (N = 64). Participation from 
reading most of the materials was 12.6%, 11.3% made 
plans to use, 7.7% talked to others, 7.1% participated in 
training, 5.5% secured or tried to secure resources, and 
2.9% received TA. Mean use among the non-AGTO 
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Table 4 Intent to treat analysis: adjusted means (standard errors) on 0-100 capacity scales' 
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59.16 (1.59) 
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0.09 
















non-tx group 


-1.08 


967 


-0.92 
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0.46 
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GTO behaviors 


53.88 (2.48) 


51.75 (1.69) 


49.62 (1.4) 


58.51 (2.53) 


57.17 (1.75) 


55.82 (1.44) 


tx group 
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non-tx group 


-2.13 


960 
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0.05 
















interaction 


0.78 


960 


0.50 


0.61 


AGTO behaviors 


38.44 (3.25) 


38.47 (2.57) 


38.5 (2.38) 


40.92 (3.31) 


42.25 (2.62) 


43.59 (2.4) 


tx group 


1.34 


958 
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non-tx group 


0.03 
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1.31 
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'Models adjusted for repeated measures and clustered matched paired randomized design. 



group was 0.48 (SD 1.09) and ranged from 1 to 5. Use 
of AGTO varied across each coalition (see Table 2). 

Prevention capacity of users and non-users in the AGTO group 

Four prevention capacity scales in the use/non-use' ana- 
lyses (Table 5) had significant difference of difference tests: 
assets efficacy (p = 0.020), GTO efficacy (p = 0.005), assets 
behaviors (p = 0.003), and AGTO behaviors (p = 0.0003). 
The average difference across two time points between 
the AGTO use and AGTO non-use groups ranged from 
6.1 to 9.3 points, resulting in a range of 12.2 to 18.6 from 
baseline to post. Figure 2 shows the model-based means 
at each time point for the four scales with significant 
difference of differences. 

Program performance and its relationship to AGTO exposure 

Noteworthy improvements for the AGTO programs were 
made in the domains of goals and process and outcome 
evaluation (Table 6). 64% of participating programs 
began a new process evaluation and 29% began a new 
outcome evaluation since receiving AGTO. Among the 
control group, only outcome evaluation had any im- 
provement. The correlation between improvement in 
each domain and the TA hours spent on that domain 
was not statistically significant, possibly due to a small 
sample size [r(7) = 0.66, p = 0.102]. Best practices, sus- 
tainability, planning, and continuous quality improve- 
ment did not improve or declined, despite having some 
TA hours spent on them. However, with greater hours, 
more improvement was noted in goals and process and 
outcome evaluation. 



Perceptions of AGTO 

The interviews with staff who had used AGTO suggested 
some contextual factors that may both facilitate (e.g., key 
leader support) and hinder (e.g., staff turnover) the imple- 
mentation of AGTO, as well as characteristics of the AGTO 
intervention that may have supported (e.g., access to a quali- 
fied TA provider) or challenged (e.g., complexity of the 
model) its implementation (Table 7). Factors that may facili- 
tate implementation include whether AGTO was integrated 
into routine activities (31%), the number of people trained 
(23%), and key leader support (15%); whereas, barriers in- 
cluded diminishing funding (31%) and staff turnover (15%). 
Interviewees reported the AGTO process was beneficial be- 
cause it offered access to a qualified TA provider (100%), a 
wide range of written support materials (92%), and peer-to- 
peer networking opportunities (85%), among other benefits. 
However, the process was also challenging for users who 
viewed it as cumbersome (54%), at times, complex (46%), 
and in competition with regular job duties (46%). Despite 
these challenges interviewees described a series of accom- 
plishments as a result of engaging in AGTO, some of which 
sustained past the end of TA. Among the most notable 
accomplishments were that programs used more sys- 
tematic processes for their work (100%), started collecting 
outcome evaluation data (100%), established stronger 
partnerships (92%), and integrated more evidence-based 
practices into their work (92%). Evaluation efforts were 
sustained by 77% of interviewees. 

Discussion 

Using the implementation research typology outlined in 
CFIR, this study evaluated the AGTO interventions 
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Table 5 Any AGTO use versus no AGTO use among participants assigned to the AGTO group: adjusted means 
(standard errors) on 0-100 capacity scales' 
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49.02 (2.09) 


user group 


4.16 


478 


2.67 


0.01 
















non-user group 


-5.13 


478 


-2.55 
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j Models adjusted for repeated measures and clustered matched paired randomized design. 



impact on the capacity of individual prevention practi- 
tioners (i.e., CFIRs individual characteristics) and the 
performance of whole programs (CFIRs implementation 
process), while accounting for several factors in CFIRs 
inner setting domain. Contrary to study hypothesis one, 
the intent-to treat analyses showed prevention practitioners 
in the control group demonstrated significant gains in as- 
sets efficacy, although the control groups GTO behaviors 
declined. There were no other significant changes over time 
for either group on any of the other measures in these 
analyses. The variation in AGTO use within the treatment 
group and use of AGTO in the control group potentially 
contributed to these non-significant findings. There are 
a few possible explanations for why individuals in the 
non-AGTO group reported use of AGTO. Since AGTO 
is a set of well-known practices for the implementation 
of high-quality prevention, it is possible that individuals 
would have been exposed to or using some of these 
practices. In addition, the AGTO book is freely available 
for purchase by anyone and in some instances control 
coalition members attended AGTO trainings. 

While intent-to-treat analyses are more traditional, 
they may be overly conservative given that GTO is a 
voluntary intervention. To help address these potential 
analytic limitations, the use/no-use' analyses were 
conducted and showed, consistent with study hypothesis 
two, significant differences between those with greater ex- 
posure to and use of AGTO. Another advantage of these 
analyses is that they avoid the potential biases of contam- 
ination (i.e., control groups using AGTO) and differential 
response rates between intervention and control groups 



common in implementation research. We also found 
that whole prevention programs in the AGTO group 
demonstrated significant gains in certain domains of 
prevention performance (i.e., performing various preven- 
tion tasks with high quality) overall and related to their use 
of TA (hypothesis three). 

In the interviews, practitioners stated that they generally 
valued AGTO, especially the TA component, and found it 
helpful to their work. However, practitioners also experi- 
enced challenges of not having enough time and resources 
to sufficiently carry out AGTO, further highlighting the 
need for TA. Prior research has indicated that these types 
of costs (i.e., time and resources) can be negatively asso- 
ciated with implementation [56]. The importance and 
contribution of TA to improving the implementation 
process is consistent with past GTO [57] and other 
studies that involve TA [58,59]. 

Although replicating earlier findings on GTOs impact 
on capacity [12,52] and program performance [12,52,53], 
this study is the first of its kind to show that an implemen- 
tation support intervention-AGTO-yielded improvements 
in practitioner capacity and consequently in program per- 
formance on a larger sample of practitioners and programs 
using a randomized controlled design. These findings sup- 
port the theory behind implementation support interven- 
tions generally and AGTO specifically, which posit that 
improving practitioner capacity to engage in the activities 
of the 10 steps of AGTO will consequently improve 
program performance. The final link in the GTO theor- 
etical chain (implementation support— >capacity—>program 
performance— »youth outcomes) is currently being tested in 
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Figure 2 Plot of the any AGTO use vs. no AGTO use among participants assigned to the AGTO group: adjusted means on 
0-100 capacity scales. 



an ongoing multi-site, randomized trial (1R01HD069427-01, 
Chinman, PI), with results expected in 2014. 

Findings from the current study show that using 
AGTO helps improve the quality of real world imple- 
mentation. There are other interventions that aim to 



build capacity-e.g., in the area of alcohol and drug preven- 
tion (e.g., Communities That Care [60] and PROmoting 
School-community-university Partnerships to Enhance 
Resilience [61]). However, unlike these interventions 
that provide support and resources to help community 



Table 6 Program performance ratings 



AGTO (Overall TA hours, % Change correlation = 0.66) n = 1 7 



Control n = 1 5 





Scores at each time 




% Change 




TA hours spent 


Scores at each time 




% Change 






Baseline 


Mid 


Post 


Baseline 
to Mid 


Mid 
to post 


Baseline 
to post 


baseline to post 


Baseline 


Mid 


Post 


Baseline 
to mid 


Mid 
to post 


Baseline 
to post 


Goals 


3.81 


3.88 


4.06 


1.64 


4.84 


6.56 


59.00 


3.62 


3.42 


3.33 


-5.5 


-2.44 


-7.8 


Best practices 


4.75 


4.81 


3.75 


1.32 


-22.08 


-21.05 


53.25 


4.69 


4.08 


3.67 


-12.98 


-10.2 


-21.86 


Planning 


4.94 


4.38 


4.25 


-11.39 


-2.86 


-13.92 


39.25 


4.62 


4.17 


4 


-9.72 


-4 


-13.33 


Process evaluation 


3.31 


3.56 


3.38 


7.55 


-5.26 


1.89 


65.75 


3.88 


3.75 


3.58 


-3.47 


-4.44 


-7.76 


Outcome evaluation 


2.78 


3.03 


3.00 


8.99 


-1.03 


7.87 


78.50 


3.08 


2.83 


3.38 


-7.92 


19.12 


9.69 


CQI 


3.69 


3.06 


3.28 


-16.95 


7.14 


-11.02 


36.50 


3.81 


2.14 


2.67 


-43.79 


24.58 


-29.97 


Sustain 


4.4 


3.94 


3.63 


-10.51 


-7.94 


-17.61 


56.00 


3.92 


2.15 


3.58 


-45.1 


66.37 


-8.66 


TOTAL 


3.95 


3.81 


3.62 


-2.77 


-3.88 


-6.76 


511.00 


3.95 


3.22 


3.46 


-18.35 


12.71 


-11.38 



k Ratings lowest score = 1, highest = 7. 
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Table 7 General categories and sub-categories of themes identified in structured interviews with AGTO programs 
(n = 13 participants) 1 



Category 


Theme 


Percent of 
interviews (n) 


Illustrative quote 


Factors that facilitated 


When AGTO is integrated into routine activities 


31% (4) 


No illustrative quote' 


AGTO implementation 


When more people are trained to use AGTO 


23% (3) 


[AGTO] spread across office of paid staff because the 
majority of staff participated in AGTO process. 




When key leaders support AGTO 


15% (2) 


No illustrative quote' 


Barriers that constrained 
AGTO implementation 


Diminishing funding climate 


31% (4) 


The funding levels for prevention just bottomed out in 
the middle of this project. 




Staff turnover due to funding 


15% (2) 


No illustrative quote' 


Challenges with the 


Can be cumbersome to implement 


54% (7) 


No illustrative quote' 


AGTO process 


A complex process 


46% (6) 


. . .It was kind of more towards the end really I kind of 
really grasped onto it. ... this is a whole other field 
that I was not in. 




Competes for time with regular job duties 


46% (6) 


If it hadn't been for ^A provider], I probably wouldn't 
have [engaged with AGTO] - she was the one that 
keep me on task because I mean this was only just 
one part of my position. 




At times, misaligned with on-the-ground 
program operations 


15% (2) 


Sometimes it was very difficult to take a program in 
Maine and have it understood by people at a great 
distance. And sometimes the suggestions that we 
would get kind of showed that they didn't quite 
get the program or an aspect of the program. 


Benefits of the 
AGTO process 


Access to a qualified TA provider 


100% (13) 


[It helped] ...to have them be able to interpret.... 
research-y language and translate that into, on the 
ground.. This is what they really mean, and this is 
what you can do with that/ 




Provided a wide range of written 
support materials 


86% (12) 


I think the manual is an absolute godsend, especially 
with the worksheets. It was really clear to understand. 
It was laid out in a way so that you could just flip to 
it and find something that might be helpful as far as 
embarking on a new project... 




Access to peer-to-peer networking 
opportunities 


85% (11) 


It was always interesting to talk to other people from 
other programs around the communities that were 
involved in AGTO. And you always got ideas from 
them. . .. They actually made me think more deeply 
about things and gave me a different perspective. 




Provided high-quality training 


69% (9) 


The training sessions that we've had have been 
extremely beneficial, and I've had nothing but 
positive feedback from our coalition members when 
we've had them. 




Proactive approach 


54% (7) 


We would have specific things that we were doing. 
It was always pretty specific. There wasn't too much 
general, though after our specific goals we be met I 
would just talk to her about just some general 
problems areas. 




Emphasized/prioritized collection and 
analysis of data 


54% (7) 


I mean, adding outcome, legitimate outcome data 
collection and looking at behavioral change was a 
significant difference because of AGTO. 




Access to larger body of experts via 
the Project Leadership Team 


31% (4) 


I really think that [the principal investigator] and 
the whole team involved in providing the support 
and TA have been superb. We're very blessed to 
have that level and quality of support because it's 
certainly made a difference here. 




Worked for participants regardless of 
baseline capacity 


31% (4) 


No illustrative quote' 
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Table 7 General categories and sub-categories of themes identified in structured interviews with AGTO programs 
(n = 13 participants) 1 (Continued) 



Program/coalition 
accomplishments 


Learned to be more systematic and 
intentional about work 


100% (13) 


[The AGTO] process, because it's rigorous and logical 
and exacting, makes you more realistic in your thinking 
about your situation .... And what you can do to affect 
[it] and what you can't do. 




Started collecting outcome evaluation data 


100% (13) 


The evaluation tools and how to use data in a way that 
informed decision making, I think, was probably the 
biggest takeaway from this experience. 




Paved the way for stronger relationships 
and partnerships 


92% (12) 


I think we better prepared ourselves even for 
approaching [partners] because we were able to make 
our programs much more defined. 




Integrated more evidence-based practices 
into their work 


92% (12) 


We're working towards having everything be 
evidence-based. 




Took inventory of needs and resources 
to better focus them 


77% (10) 


[We learned]... what we have available in terms of 
partner organizations, their resources, the coalition's 
resources, and . . . how can we focus those so 
[they]... produce change. 




Improved youth voice in program/ 
coalition activities 


62% (8) 


Youth are involved in program planning, 
implementation, and evaluation. 




Improved communication about 
program/ coalition 


38% (5) 


No illustrative quote' 


Aspects of AGTO sustained 
after the project ended 


Continued to evaluate their program/coalition 


77% (10) 


We've had the physical presence of people who are 
keeping [evaluation efforts] going. 




Continued to work all 10 steps of AGTO 


38% (5) 


AGTO becomes endemic to the work. Even though 
TA ends you don't unlearn the skills and knowledge 
developed. 




Generalized AGTO to other activities/topic areas 


38% (5) 


No illustrative quote' 




Continued to use manuals and other 
written support materials 


38% (5) 


Just a couple of weeks ago ...I was getting the 
Getting To Outcomes with Developmental Assets 
[manual] out. I was looking through it and I see 
things that I haven't seen before. 




Continued to use logic model to guide 
program/coalition work 


23% (3) 


No illustrative quote' 




Not able to sustain AGTO 


23% (3) 


I think that support [TA] was crucial and, without it, 
it really hasn't-it's been hard to sustain. 



'Cells without illustrative quotes were left blank because there was no single quote that illustrated the theme, but was the result of a much longer narrative that includes 
exchanges between the interviewer and interviewee. 



practitioners select and begin implementing evidence- 
based prevention programs, AGTO engages existing 
programs (whether evidence-based or home-grown) and 
the leadership of organizations that house these programs 
to improve their quality through support with no additional 
financial resources for implementation. As such, the 
goal of the AGTO intervention is to help the leadership 
of community organizations to integrate the practices 
AGTO targets into routine operations. 

The methods used in the current study also have im- 
plications for broader implementation research using 
the CFIR. The study used measures to assess the quality 
of the implementation process as well as the relevant 
capacity of practitioners (i.e., individuals involved) that 
could be adapted to many other interventions, and exist 
already for interventions in the areas of substance abuse 
prevention, positive youth development, prevention of 
teen pregnancy and sexually transmitted infections, and 
homelessness prevention. These measures were used in 



an randomized controlled trial and our analytic approach 
addressed certain challenges of conducting research on 
real-world implementation (e.g., the turnover of member- 
ship at participating coalitions, differential response rates 
between intervention and control groups). More innovative 
methods and more acceptance of nontraditional research 
methods will be needed to capture the complex, multi-level 
and transient status of constructs in the real-world as the 
study of implementation evolves. 

Limitations 

The interviews to gather perceptions of AGTO were done 
with a convenience sample, which could have introduced 
bias into those findings. This was in part a result of 
diminishing budgets for prevention in Maine, which 
caused a sizable dropout of programs between baseline 
and post (although equivalent across groups). A sample of 
32 programs with complete program performance ratings 
is modest. Therefore the percent change analyses should 
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be interpreted with caution. Future studies that recruited 
even larger numbers of schools, communities, or coalitions 
would provide the opportunity to use inferential statistics 
to better test the relationships between TA and various 
prevention tasks. Further, TA hours, even by domain, is 
not the only important measure of TA; quality of the TA, 
timing about when it was given (e.g., planning before 
evaluation), and type of program are other factors to be 
explored in future research [62,63]. Also, because the use 
versus non-use groups were not randomly assigned, there 
may have been selection biases within use versus non-use 
analyses-e.g., practitioners who engaged in the interven- 
tion may have been more likely to improve than those 
who did not. However, this bias may be mitigated some- 
what because the capacity scores with significant group 
(user versus non-user) X time (pre-mid-post) interactions 
were not significantly different at baseline among the use 
versus non-use groups. It is important to note, capacity- 
building interventions like AGTO are voluntary and thus 
this analysis represents a real-world test. 

Conclusions 

The types of programs and practitioners AGTO targeted 
in this study are typical of prevention efforts in many 
communities. Despite challenges, AGTO helped programs 
that made use of it make critical improvements in the qual- 
ity of their work. Implementation support interventions like 
AGTO can help ensure investments in real world preven- 
tion programming are conducted with high quality and 
consequently yield positive outcomes. 

Endnotes 

a Getting To Outcomes and GTO are trademarks reg- 
istered by University of South Carolina and RAND. 
b US Census 2000. 

c According to the 2006 Maine Youth Drug and 
Alcohol Use Survey (www.maine.gov/dhhs/osa/data/ 
mydaus/mydaus2006.htm). 

d Nine programs ended between the time the study was 
planned and begun, leaving only 51 programs at baseline. 
19 more programs ended between baseline and post. 

Additional file 

Additional file 1: Table SI. CONSORT Checklist (Version 2010). 



Abbreviations 

AGTO: Assets getting to outcomes; CFIR: Comprehensive framework for 
implementation research; GTO: Getting to outcomes; PABAK: 
Prevalence-adjusted bias-adjusted kappa; TA: Technical assistance. 

Competing interests 

The authors declare that they have no competing interests. 



Authors' contributions 

AP carried out the qualitative interviews, qualitative analysis of interviews 
and drafted the manuscript. MS, SP and PM designed and performed the 
statistical analyses. PS participated in the drafting of the manuscript. JA, MC, 
and PE conceived of the study, and participated in its design and 
coordination and helped to draft the manuscript. All authors read and 
approved the final manuscript. 



Authors' information 

Joie Acosta is a Behavioral Scientist at the RAND Corporation and specializes in 
research and evaluation to improve the implementation of evidence-based 
positive youth development, substance abuse and mental health prevention 
and treatment programs. From this work, she co-authored reports on building 
capacity for outcomes measurement among human service organizations and 
improving the implementation of social services and behavioral health care. 
Matthew Chinman is a Senior Behavioral Scientist at the RAND Corporation, 
whose recent focus has been to develop strategies to enhance the capacity 
of community-based practitioners. He co-developed Getting To Outcomes® 
(GTO), a model designed to help community organizations to better plan, 
implement, and self-evaluate programs across a number of domains. He has 
led the successful testing of the several Getting To Outcomes-based guides 
in the areas of substance abuse prevention, underage drinking prevention, 
and youth development. 

Patricia Ebener is a senior researcher with RAND's Drug Policy Research 
Center and a senior survey director with RAND's Survey Research Group. She 
has collaborated on community based participatory research and quality of 
prevention and treatment services projects for over twenty years. 
Patrick Malone, a quantitative and social psychologist, is an Associate 
Professor of Psychology at the University of South Carolina. He specializes in 
methods for latent variable modeling and longitudinal analysis for 
adolescent health risk behaviors. 

Susan Paddock is a Senior Statistician at the RAND Corporation. Dr. 
Paddock's research interests include statistical methods development and 
statistical applications to substance abuse treatment studies. 
Andrea Phillips is a Project Associate at the RAND Corporation. 
Peter C. Scales is Senior Fellow at Search Institute. For nearly four decades, 
he has been widely recognized nationally, and more recently, internationally, 
for his research, direct service work, and comprehensive writings on 
promoting positive child and youth development. He is the world's leading 
researcher on the Developmental Assets approach to positive development, 
has led more than 15 large-scale or nationally representative studies, and 
international studies in a dozen countries on five continents, and authored 
or co-authored more than 250 publications, including 10 books, and 75 
peer-reviewed journal articles and book chapters. 
Mary Ellen Slaughter is a statistical/quantitative analyst at RAND with 
statistical expertise in regression analysis, categorical data analysis, survival 
analysis, and multivariate analysis. She also has experience with cost- 
effectiveness analysis, large health care databases, geographic information 
systems, and is proficient in SAS and R statistical programming. 



Funding 

Enhancing Prevention Capacity with Developmental Assets and Getting to 
Outcomes, 1 R01 DA023277-01 A2, National Institute on Drug Abuse (NIH). 

Author details 

] RAND Corporation, 4570 Fifth Avenue, Pittsburgh PA 15213, USA. University 
of South Carolina, 920 Sumter St, Columbia SC 29208, USA. 3 Search Institute, 
615 First Avenue NE, Suite 125, Minneapolis MN 55413, USA. 

Received: 12 April 2013 Accepted: 31 July 2013 
Published: 7 August 2013 



References 

1 . Miller T, Hendrie D: Substance abuse prevention dollars and cents: A 
cost-benefit analysis (DHHS Pub. No. (SMA) 07-4298). Rockville, MD: Center for 
Substance Abuse Prevention, Substance Abuse and Mental Health Services 
Administration; 2009. 

2. Backer TE: Finding the balance: program fidelity and adaptation in 
substance abuse prevention: CSAP. 2001. 



Acosta et al. Implementation Science 2013, 8:87 
http://www.implementationscience.eom/content/8/1/87 



3. Ennett ST, Ringwalt CL, Thome J, Rohrbach LA, Vincus A, Simons-Rudolph A, 
et al: A comparison of current practice in school-based substance abuse 
prevention programs with meta-analysis findings. Prev Sci 2003, 4:1-14. 

4. Cantor D, Crosse S, Hagen CA, Mason MJ, Siler AJ, Von Glatz A: Wide Scope, 
Questionable Quality: Drug and Violence Prevention Efforts in American Schools. 
Report on the Study on School Violence and Prevention. In. Washington, DC: U.S. 
Department of Education, Planning and Evaluation Service; 2001. 

5. Crosse S, Burr M, Cantor D, Hagen CA, Hantman I: Wide Scope, Questionable 
Quality: Drug and Violence Prevention Efforts in American Schools. Report on 
the Study on School Violence and Prevention. In. Washington, DC: U.S. 
Department of Education, Planning and Evaluation Service; 2001. 

6. Reuter P, Timpane JM: Assessing options for the safe and drug free schools 
and communities act. Santa Monica, CA: RAND; 2001. No. MR-1328. 

7. Silvia ES, Thome J: School-based drug prevention programs: A longitudinal 
study in selected school districts. In. North Carolina: Research Triangle 
Institute; 1997. 

8. U.S. Department of Education OoP, Evaluation and Policy Development, 
Policy and Program Studies Service: Prevalence and Implementation Fidelity 
of Research-Based Prevention Programs in Public Schools: Final Report. 
Washington, D.C; 2011. 

9. Wandersman A, Florin P: Community interventions and effective 
prevention: bringing researchers/evaluators, funders and practitioners 
together for accountability. Am Psychol 2003, 58(6/7):441-448. 

10. Green L: From research to 'Best Practices' in other settings and 
populations. Am J Health Behav 2001, 25(3):165-178. 

1 1 . Livet M, Wandersman A: Organizational functioning: facilitating effective 
interventions and increasing the odds of programming success. In 
Empowerment evaluation in practice. Edited by Fetterman DM, Wandersman 
A. New York, NY: Guilford; 2005:123-154. 

12. Chinman M, Hunter S, Ebener P, Paddock S, Stillman L, Imm P, Wandersman 
A: The Getting To Outcomes demonstration and evaluation: an 
illustration of the prevention support system. Am J Community Psychol 
2008,41:206-224. 

13. Chinman M, Tremain B, Imm P, Wandersman A: Strengthening prevention 
performance using technology: a formative evaluation of interactive 
Getting To OutcomesTM. Am J Orthopsychiatry 2009, 79:469-481. 

14. Fishbein M, Ajzen A: Beliefs, attitudes, intentions, and behavior: An 
introduction to theory and research. Reading, MA: Addison-Wesley; 1975. 

15. Ajzen I, Fishbein M: Attitude-behavior relations: a theoretical analysis and 
review of empirical research. Psychol Bull 1977, 84:888-918. 

16. Bandura A: Health promotion by social cognitive means. Health Educ 
Behav 2004, 3 1(2): 143- 164. 

17. Fishbein M, Ajzen I: Atitudes toward objects as predictive of single and 
multiple behavioral criteria. Psychol Rev 1974, 81:59-74. 

18. Chinman M, Acosta J, Ebener P, Burkhart Q, Clifford M, Corsello M, Duffy T, 
Hunter S, Jones M, Lahti M, et al: Establishing and evaluating the key 
functions of an interactive systems framework based on Assets-Getting 
To Outcomes. Am J Commun Psychol 2012. http://rd.springer.com/article/ 

1 0.1 007%2Fs1 0464-01 2-9504-z. 

19. Benson PL, Scales PC, Leffert N, Roehlkepartain ER: A fragile foundation: The 
state of developmental assets among American youth. Minneapolis, MN: 
Search Institute; 1999. 

20. Leffert N, Benson PL, Scales PC, Sharma AR, Drake DR, Blyth DA: 
Developmental assets: measurement and prediction of risk behaviors 
among adolescents. Appl Dev Sci 1998, 2:209-230. 

21. Scales PC, Benson PL, Leffert N, Blyth DA: Contribution of developmental 
assets to the prediction of thriving among adolescents. Appl Dev Sci 
2000, 4:27-46. 

22. Fisher D, Imm PS, Chinman M, Wandersman A: Getting To Outcomes with 
Developmental Assets: Ten steps to measuring success in youth programs and 
communities. Minneapolis, MN: Search Institute; 2006. 

23. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: 
Fostering implementation of health services research findings into 
practice: a consolidated framework for advancing implementation 
science. Implement Sci 2009, 4:50-65. 

24. Rohrbach LA, D'Onofrio CN, Backer TE, Montgomery SB: Diffusion of 
school-based substance abuse prevention programs. Am Behav Sci 1996, 
39:919-934. 

25. Rohrbach LA, Graham JW, Hansen WB: Diffusion of a school-based 
substance abuse prevention program: predictors of program 
implementation. Prev Med 1993, 22(2):237-260. 



Page 15 of 16 



26. McCormick LK, Steckler AB, McLeroy KR: Diffusion of innovations in 
schools: a study of adoption and implementation of school-based 
tobacco prevention curricula. Am J Health Promot 1995, 9:210-219. 

27. Steckler AB, McLeroy KR, Goodman RM, McCormick L, Bird ST: Toward 
integrating qualitative and quantitative methods: an introduction. 
Health Educ Q 1 992, 19:1-8. 

28. Parcel GS, Erikson MP, Lovato CY, Gottlieb NH, Brink SG, Green LW: The 
diffusion of school-based tobacco-use prevention program: project 
description and baseline data. Health Educ Res 1989, 4:1 1 1-124. 

29. Cividin TM, Ottoson JM: Linking reasons for continuing professional 
education participation with post-program application. J Contin Educ 
Health Professions 1997, 17:46-55. 

30. Ottoson JM: After the applause: exploring multiple influences on 
application following an adult education program. Adult Educ Quart 

1997, 47:92-107. 

31. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: 
Fostering implementation of health services research findings into 
practice: a consolidated framework for advancing implementation 
science. Implement Sci 2009, 4:1. 

32. Murray DM: Design and analysis of group-randomized trials. Oxford, UK: Press 
OU; 1998. 

33. Battelle: Annual findings report: Drug-free communities support program 
national evaluation. Washington, DC: Executive Office of the President Office 
of Administration Office of National Drug Control Policy; 2008. 

34. Butterfoss FD, Goodman RM, Wandersman A: Community coalitions for 
prevention and health promotion. Health Educ Res 1993, 8:315-330. 

35. Ellickson PL, Bell RM: Drug prevention in junior high: a multi-site 
longitudinal test. Science 1990, 247:1265-1372. 

36. Botvin GJ, Baker E, Dusenbury L, Botvin EM, DiazT: Long-term follow-up 
results of a randomized drug abuse prevention trial in a white 
middle-class population. JAMA 1995, 273(1 4):1 106-1 1 12. 

37. Jemmott JB, Jemmott L, Spears H, Hewitt N, Cruz-Collins M: Self-efficacy, 
hedonistic expectancies, and condom-use intentions among inner-city 
Black adolescent women: a social cognitive approach to AIDS risk 
behavior. J Adolescent Health 1 992, 1 3:5 1 2-5 1 9. 

38. Jemmott JB, Jemmott LS, Fong GT: Reductions in HIV risk-associated 
sexual behaviors among Black male adolescents: effects of an AIDS 
prevention intervention. Am J Public Health 1992, 82:372-377. 

39. Jemmott JB, Jemmott LS, Fong GT, McCaffree K: Reducing HIV 
risk-associated sexual behavior among African American adolescents: 
Testing the generality of intervention effects. Am J Community Psychol 
1999, 27:161-187. 

40. Chinman M, Hannah G, Wandersman A, Ebener P, Hunter S, Imm P, Sheldon 
J: Developing a community science research agenda for building 
community capacity for effective preventive interventions. Am J 
Community Psychol 2005, 3/4:143-157. 

41. Mitchell R, Stone-Wiggins B, Stevenson JF, Florin P: Cultivating capacity: 
outcomes of a statewide support system for prevention coalitions. 

J Prev Interv Community 2004, 27:67-87. 

42. O'Donnell LO, Scattergood P, Adler M, San Doval A, Barker M, Kelly JA, et al: 
The role of technical assistance in the replication of effective HIV 
interventions. AIDS Educ Prev 2000, 1 2:99-1 1 1 . 

43. Stevenson JF, Florin P, Mills DS, Andrade M: Building evaluation capacity in 
human service organizations: a case study. Eval Program Plann 2002, 
25(3):233-243. 

44. Kitson A, Harvey G, McCormack B: Enabling the implementation of 
evidence based practice: a conceptual framework. Qual Health Care 

1998, 7:149-158. 

45. Rycroft-Malone J, Harvey G, Seers K, Kitson A, McCormack B, Titchen A: An 
exploration of the factors that influence the implementation of evidence 
into practice. J Clin Nurs 2002, 13:913-924. 

46. Stetler CB, Legro MW, Rycroft-Malone J, Bowman C, Curran G, Guihan M, 

et al: Role of 'external facilitation' in implementation of research findings: 
a qualitative evaluation of facilitation experiences in the Veterans Health 
Administration. Implement Sci 2006, 23. 

47. Weiss E, Miller-Anderson R, Lasker R: Making the most of collaboration: 
exploring the relationship between partnership synergy and partnership 
functioning. Health Educ Behav 2002, 29:683-698. 

48. Lehman WEK, Greener JM, Simpson DD: Assessing organizational 
readiness for change. J Subst Abuse Treat 2002, 22(4):1 97-209. 

49. Hall GE, Hord SM: Implementing change. Boston, MA: Allyn and Bacon; 2001. 



Acosta et al. Implementation Science 2013, 8:87 
http://www.implementationscience.eom/content/8/1/87 



Page 16 of 16 



50. Hall GE, Loucks SF, Rutherford WL, Newlove BW: Levels of use of the innovation: 
a framework for analyzing innovation adoption. J Teach Educ 1975, 26:52. 

51. Hall GE, Loucks SF: A developmental model for determining whether the 
treatment is actually implemented. Am Educ Res J 1977, 14(3):263-276. 

52. Chinman M, Acosta J, Ebener P, Burkhart Q, Clifford M, Corsello M, Duffy T, 
Hunter S, Jones M, Lahti M, et al: Intervening with practitioners to 
improve the quality of prevention: One year findings from a randomized 
trial of Assets-Getting To Outcomes. J Prim Prev. In press. 

53. Chinman M, Tremain B, Imm P, Wandersman A: Strengthening prevention 
performance using technology: a formative evaluation of interactive 
Getting To Outcomes. Am J Orthopsychiatry 2009, 79:469-481. 
PMC2859836. 

54. Bryt T, Bishop J, Carlin JB: Bias, prevalence and kappa. J Clin Epidemiol 
1993, 46(5):423-429. 

55. Lincoln YS, Guba EG: Naturalistic inquiry. In. Beverly Hills, CA: Sage; 1985. 

56. Graham ID, Logan J: Innovations in knowledge transfer and continuity of 
care. Can J Nurs Res 2004, 36(2):89-103. 

57. Hunter SB, Chinman M, Ebener P, Imm P, Wandersman A, Ryan G: Technical 
assistance as a prevention capacity-building tool: a demonstration using 
the Getting To Outcomes framework. Health Educ Behav 2009, 36:810-828. 

58. Keener DC: Toward a science of capacity building: An examination of 
technical assistance following a training program for prevention professionals. 
In. Columbia, SC, USA: University of South Carolina; 2007. 

59. Feinberg ME, Ridenour TA, Greenberg MT: The longitudinal effect of 
technical assistance dosage on the functioning of communities that care 
prevention boards in Pennsylvania. J Prim Prev 2008, 29(2):1 45-1 65. 

60. Hawkins JD, Oesterle S, Brown EC, Arthur MW, Abbott RD, Fagan AA, 
Catalano RF: Results of a type 2 translational research trial to prevent 
adolescent drug use and delinquency: a test of communities that care. 
Arch Pediatr Adolesc Med 2009 September, 163(9)789-798. 

61 . Spoth R, Redmond C, Shin C, Greenberg M, Clair S, Feinberg M: Substance-use 
outcomes at 18 months past baseline: the PROSPER Community-University 
partnership trial. Am J Prev Med 2007 May, 32(5)395-402. 

62. Wandersman A: Four keys to success (theory, implementation, 
evaluation, and resource/system support): high hopes and challenges in 
participation. Am J Community Psychol 2009, 43:3-21 . 

63. Wandersman A, Chien VH, Katz J: Toward an evidence-based system for 
innovation support for implementing innovations with quality: tools, training, 
technical assistance, and quality assurance/quality improvement. In Cultivating 
communities of practice. Edited by Wenger E, McDermott R, Snyder WM. Boston, 
MA: Harvard Business School Press; 2002. In press. 



doi:1 0.1 186/1 748-5908-8-87 

Cite this article as: Acosta et al.: An intervention to improve program 
implementation: findings from a two-year cluster randomized trial of 
Assets-Getting To Outcomes. Implementation Science 2013 8:87. 



Submit your next manuscript to BioMed Central 
and take full advantage of: 

• Convenient online submission 

• Thorough peer review 

• No space constraints or color figure charges 

• Immediate publication on acceptance 

• Inclusion in PubMed, CAS, Scopus and Google Scholar 

• Research which is freely available for redistribution 



Submit your manuscript at (^\ RiftMM i rpntral 

www.biomedcentral.com/submit \^ ™omea centra I 



